CS224N Natural Language Processing with Deep Learning Assignment 2

距离上次更新已经 1391 天了，文章内容可能已经过时。

课程主页：https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/

视频地址：https://www.bilibili.com/video/av46216519?from=search&seid=13229282510647565239

这里回顾CS224N Assignment 2的内容，Assignment 1比较基础，这里从略。

1.Understanding word2vec

(a)

注意只有当$w=o$时，我们才有$y_w=1$，其余情形均为$0$，所以

$$-\sum _{w\in Vocab}{y}_w\log \left({\hat{y}}_w\right)=-\log \left({\hat{y}}_o\right)$$

(b)

$$\begin{array}{rl}{J}_{\text{naive-softmax}}({\mathit{v}}_c,o,\mathit{U})& =-\log P(O=o|C=c)\\ & =-\log \frac{\exp \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right)}{\sum _{w\in \mathrm{Vocab}}\exp \left({\mathit{u}}_{\mathit{w}}^{\mathrm{\top }}{\mathit{v}}_c\right)}\\ & =-u_o^{\mathrm{\top }}{v}_c+\log \sum _{w\in \mathrm{Vocab}}\exp \left({\mathit{u}}_{\mathit{w}}^{\mathrm{\top }}{\mathit{v}}_c\right)\end{array}$$

所以

$$\begin{array}{rl}\frac{\partial J_{\text{naive-softmax}}({\mathit{v}}_c,o,\mathit{U})}{\partial v_c}& =-u_o+\sum _{o\in \mathrm{Vocab}}\frac{\exp (u_o^{\mathrm{\top }}{v}_c)}{\sum _{w\in \mathrm{Vocab}}\exp \left({\mathit{u}}_{\mathit{w}}^{\mathrm{\top }}{\mathit{v}}_c\right)}\frac{\partial (u_o^{\mathrm{\top }}{v}_c)}{\partial v_c}\\ & =-u_o+\sum _{o\in \mathrm{Vocab}}P(O=o|C=c)u_o\\ & =-Uy+U\hat{y}\\ & =U(\hat{y}-y)\end{array}$$

(c)

$$\begin{array}{rl}\frac{\partial J_{\text{naive-softmax}}({\mathit{v}}_c,o,\mathit{U})}{\partial u_w}& =-v_c{1}_{\{w=o\}}+\frac{\exp (u_w^{\mathrm{\top }}{v}_c)}{\sum _{w\in \mathrm{Vocab}}\exp \left({\mathit{u}}_{\mathit{w}}^{\mathrm{\top }}{\mathit{v}}_c\right)}\frac{\partial (u_w^{\mathrm{\top }}{v}_c)}{\partial u_w}\\ & =-v_c{1}_{\{w=o\}}+P(O=w|C=c)v_c\\ & =v_c({\hat{y}}_w-y_w)\end{array}$$

(d)

计算雅克比矩阵可得

$$\begin{array}{rl}\frac{\partial \sigma (x_i)}{\partial x_j}& =\sigma (x_i)(1-\sigma (x_i))1_{\{i=j\}}\end{array}$$

所以

$$\frac{\partial \sigma (x)}{\partial x}=\text{diag}(\sigma (x)(1-\sigma (x)))$$

(e)

注意$o$不属于$\{1,\dots ,K\}$

$$\begin{array}{rl}\frac{\partial J_{\text{neg-sample}}(v_c,o,U)}{\partial v_c}& =-\frac{\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right)(1-\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right))}{\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right)}{u}_o-\sum _{k=1}^K\frac{\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c)(1-\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c))}{\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c)}(-u_k)\\ & =-(1-\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right))u_o+\sum _{k=1}^K(1-\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c))u_k\\ \frac{\partial J_{\text{neg-sample}}(v_c,o,U)}{\partial u_o}& =-\frac{\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right)(1-\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right))}{\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right)}{v}_c\\ & =-(1-\sigma \left({\mathit{u}}_o^{\mathrm{\top }}{\mathit{v}}_c\right))v_c\\ \frac{\partial J_{\text{neg-sample}}(v_c,o,U)}{\partial u_k}& =-\frac{\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c)(1-\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c))}{\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c)}(-v_c)\\ & =(1-\sigma (-{\mathit{u}}_k^{\mathrm{\top }}{\mathit{v}}_c))v_c\end{array}$$

原始的损失函数中需要求指数和，很容易溢出，要进行处理，但是负采样的损失函数就没有这个问题。

(f)

$$\begin{array}{rl}\partial {\mathit{J}}_{\text{skip-gram }}({\mathit{v}}_c,w_{t-m},\dots w_{t+m},\mathit{U})/\partial \mathit{U}& =\sum _{\genfrac{}{}{0ex}{}{-m\le j\le m}{j\ne 0}}\partial \mathit{J}({\mathit{v}}_c,w_{t+j},\mathit{U})/\partial \mathit{U}\\ \partial {\mathit{J}}_{\text{skip-gram }}({\mathit{v}}_c,w_{t-m},\dots w_{t+m},\mathit{U})/\partial {\mathit{v}}_{\mathit{c}}& =\sum _{\genfrac{}{}{0ex}{}{-m\le j\le m}{j\ne 0}}\partial \mathit{J}({\mathit{v}}_c,w_{t+j},\mathit{U})/\partial {\mathit{v}}_{\mathit{c}}\\ \partial {\mathit{J}}_{\text{skip-gram }}({\mathit{v}}_c,w_{t-m},\dots w_{t+m},\mathit{U})/\partial {\mathit{v}}_{\mathit{w}}& =\sum _{\genfrac{}{}{0ex}{}{-m\le j\le m}{j\ne 0}}\partial \mathit{J}({\mathit{v}}_c,w_{t+j},\mathit{U})/\partial {\mathit{v}}_{\mathit{w}}\end{array}$$

2.Implementing word2vec

(a)

sigmoid

def sigmoid(x):
    """
    Compute the sigmoid function for the input here.
    Arguments:
    x -- A scalar or numpy array.
    Return:
    s -- sigmoid(x)
    """

    ### YOUR CODE HERE
    s = 1 / (1 + np.exp(-x))

    ### END YOUR CODE

    return s

naiveSoftmaxLossAndGradient

注意这里的矩阵是第一题矩阵的转置。

### YOUR CODE HERE

### Please use the provided softmax function (imported earlier in this file)
### This numerically stable implementation helps you avoid issues pertaining
### to integer overflow. 
'''
    centerWordVec: 1 * d
    outsideVectors: n * d
    '''
#1 * n
vec = centerWordVec.dot(outsideVectors.T)
#1 * n
prob = softmax(vec)
loss = -np.log(prob[outsideWordIdx])
#1 * d
gradCenterVec = -outsideVectors[outsideWordIdx] + prob.dot(outsideVectors)
#n * d
gradOutsideVecs = prob.reshape(-1, 1).dot(centerWordVec.reshape(1, -1))
#n * d
gradOutsideVecs[outsideWordIdx] -= centerWordVec
### END YOUR CODE

negSamplingLossAndGradient

### YOUR CODE HERE

### Please use your implementation of sigmoid in here.
'''
centerWordVec: 1 * d
outsideVectors: n * d
'''
#1 * m
vec = centerWordVec.dot(outsideVectors[indices].T)
vec[1:] *= -1
sig = sigmoid(vec)
tmp = np.log(sig)
loss = -tmp[0] - np.sum(tmp[1:])
#1 * m
t1 = 1 - sig
gradCenterVec = t1.dot(outsideVectors[indices]) - 2 * t1[0] * outsideVectors[outsideWordIdx]
#累加
gradOutsideVecs = np.zeros_like(outsideVectors)
gradOutsideVecs[outsideWordIdx] += -t1[0] * centerWordVec
for i in range(K):
    k = negSampleWordIndices[i]
    gradOutsideVecs[k] += t1[i + 1] * centerWordVec
### END YOUR CODE

(b)

### YOUR CODE HERE
loss, grad = f(x)
x -= step * grad

### END YOUR CODE